Assessing the Quality of AI Explanations of Atrial Fibrillation: Comparative Analysis of ChatGPT and Google Gemini

Introduction

Atrial fibrillation (AF) is a common arrhythmia with increasing prevalence, particularly among older adults, individuals who consume alcohol, and smokers [1]. AF significantly elevates the risk of stroke, ischemic events, and hospitalizations. However, lifestyle modifications, such as regular exercise and abstinence from alcohol and smoking, can help reduce AF risk [1]. When the diagnosis of AF is confirmed, medications and interventions can control heart rate, rhythm, and clot formation. Given these risks and treatment complexities, effective patient education for AF prevention and management is essential.

Patient education plays a crucial role in improving adherence to treatment plans and lifestyle modifications. Low health literacy has been associated with increased health care use and noncompliance with medical recommendations [2]. Educational interventions directed toward patients with chronic heart failure have been shown to enhance quality of life and medication adherence [3-5]. Similarly, digital education tools have demonstrated benefits in other medical contexts; for example, postoperative knee pain management following total knee replacement was improved with a mobile app delivering consistent daily education [6]. Despite adherence improvement, patient education must be assessed for veracity and utility. For instance, 2 studies on the quality of YouTube videos for coronary artery disease and colorectal cancer received low ratings in terms of accuracy and usefulness [7,8].

With advancements in artificial intelligence (AI), platforms such as ChatGPT have become increasingly used for patient education. Studies have shown that AI platforms can improve patient knowledge regarding prostate cancer, diabetes, and hypertension by providing accessible and accurate medical information [9-11]. However, despite these promising results, there are still gaps in the clarity, reliability, and perception of AI-generated health information that need to be addressed. When assessing information on chronic kidney disease, AI-generated responses were found to have some inaccuracies when compared with clinical guidelines [12]. This gap creates challenges for patients and physicians who have different expectations regarding the disease in question and therapeutic goals.

While several studies have explored AI’s role in predicting AF based on patient risk factors, fewer studies have examined the quality of AI-generated patient education for AF. One study assessed AI-generated responses to general AF-related questions, but it did not specifically evaluate patient-focused explanations or include nonmedical professional reviewers [13-16]. Another study demonstrated how AI was able to adapt responses for physicians or patients, but it did not compare responses across multiple AI platforms or analyze perceptual differences between medical professionals and laypeople [17]. In this study, we aimed to address these gaps by assessing the clarity, comprehensiveness, and accuracy of AI-generated patient education on AF. Specifically, we compared responses from ChatGPT and Google Gemini, analyzing their abilities to provide clear and accurate information to patients with AF. Additionally, 2 comparison groups, consisting of cardiologists and cardiology fellows and nonmedical professionals, were asked to review the responses. To our knowledge, this is the first study to directly compare AI-generated AF education across multiple AI platforms and distinct user groups (cardiologists vs nonmedical professionals). This study will provide valuable information about strengths and weaknesses of each AI platform, as well as the interpretation of AI-generated responses in a diverse patient population.

Methods

Study Design

A comparative analysis was conducted on September 6, 2024, using ChatGPT (GPT-4; OpenAI) and Google Gemini 2.5 Flash. Both AI platforms generated conversational responses to user queries. The prompt, “Explain atrial fibrillation and how to treat it to a patient,” was entered into each AI platform. The AI-generated responses were emailed to both participant groups.

Each participant received an email with the 2 prompts, instructions on how to answer the prompts, and the survey. Both prompts were included in the same email, with ChatGPT labeled as prompt #1 and Google Gemini listed as prompt #2 (Textbox 1). Participants completed the survey twice: once for the ChatGPT response and once for the Google Gemini response. Each response to the survey was anonymous. Although ChatGPT is available as an extension on Doximity, we used the standard ChatGPT interface, as it is the most likely source for patient education. For further explanation of the study design, please refer to Figure 1. This study was conducted in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines for observational studies.

‎

Assessment Tools

A survey was developed using Google Forms to evaluate responses. Clarity and readability were assessed using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P), while comprehensiveness was measured using the DISCERN criteria [18,19]. The initial survey contained 35 questions, but 6 (17.1%) duplicates between the 2 criteria were removed, resulting in a final set of 29 (82.9%) Likert scale questions.

When applying the PEMAT-P and DISCERN criteria to assess the quality of AI-generated responses, the initial response to the prompt in each AI platform was evaluated. Rather than engaging in a conversation with the AI platform, we took the initial response as a stand-alone source of patient education that a patient might receive from their cardiologist. This allowed us to assess both the content and the structure of each initial response using validated criteria.

Participant Selection

Two groups completed the survey: (1) a cardiologist group consisting of practicing cardiologists and cardiology fellows and (2) a nonmedical professional group, defined as individuals with a 4-year college degree who were not employed in the medical field. Participants from both groups were recruited via standardized emails, including reminder messages, and completed the survey between September and October 2024. The cardiologists and cardiology fellows were recruited through departmental mailing lists and professional contacts in the cardiology communities of both New Jersey and New York. The nonmedical professionals were recruited through community contacts and professional outreach in the United States and the United Kingdom. In total, 11 cardiologists across the United States and 17 nonmedical professionals from the United States and the United Kingdom filled out the survey.

Statistical Analysis

Means of survey responses were calculated for each subgroup (cardiologists, nonmedical professionals, ChatGPT, and Google Gemini). Comparisons were conducted using the Wilcoxon signed-rank test, a nonparametric test appropriate for ordinal data. A P value of <.05 was considered statistically significant.

Given the modest sample size of this study, no multivariable adjustments for potential confounders were performed. To minimize bias, all participants reviewed the same standardized AI-generated responses, and responses were analyzed in aggregate. All identifiable demographic information was removed from the data. Effect modifiers, such as reviewer expertise (medical vs nonmedical), were examined by stratified analysis of survey ratings across groups. Given that there was only 1 cardiologist who missed a response, there were no data adjustments to accommodate missing data.

Study Inclusion and Exclusion Criteria

The inclusion and exclusion criteria are presented in Textbox 2.

Ethical Considerations

This study did not meet the definition of human research as defined by Stony Brook Human Research Protection Program Standard Operating Procedure, section 1.5, as no patient information was collected. In accordance with section 3.8.5, no private information was collected, as all data were derived from publicly available AI platforms. Per section 3.4, the investigator determined that this study did not involve human subjects and did not require institutional review board approval. On the basis of the abovementioned criteria, this study falls outside the scope of research requiring institutional review board approval. This study has been reported in accordance with STROBE guidelines. Both study cohort participants were voluntary, anonymous, and were not compensated.

Results

Comparison Between Google Gemini and ChatGPT Responses

A total of 55 survey responses were collected. There were 29 (52.7%) responses to the ChatGPT-generated prompt and 26 (47.3%) responses to the Google Gemini–generated prompt (Table 1). As shown in Table 2, there were no significant differences between responses to ChatGPT and Google Gemini across all questions. From the DISCERN criteria, both platforms received low ratings for questions addressing risks of treatment, changes in quality of life, and what happens if no treatment was used (Table 2). From the PEMAT-P questions, questions assessing the use of visual aids, such as charts and graphs, had low ratings from both platforms (Table 3).

Comparison Between Cardiologists and Nonmedical Professionals

Of the 55 survey responses collected, 34 (61.8%) were from noncardiologists and 21 (38.2%) were from the cardiologist group (Table 4). Table 3 shows that the cardiologist group had a significant difference in ratings of bias when compared with the nonmedical group (P=.03). Additionally, the nonmedical group provided a significantly higher rating when describing what would happen if there was no treatment for AF (P=.005; Table 3). The cardiologist group had significantly lower ratings for shorter sections of the prompt (P=.04), informative headers (P=.001), and the use of visual cues (P<.001; Table 3). Additionally, cardiologists rated the responses significantly lower in key areas, including the use of shorter sections (P=.04), informative headers (P=.001), and visual cues (P<.001).

The nonmedical group showed lower ratings when compared with the cardiology group for identifying actions (P=.001) and describing how those actions could be taken (P=.04), as well as accessibility to links for further reading (P=.006). Nonmedical professionals also provided significantly higher ratings for explanations of what would happen if AF remained untreated (P=.005). Compared with cardiologists, the nonmedical group gave significantly lower ratings for identifying actionable steps (P=.001), explaining how to take those actions (P=.04), and accessibility to links for further reading (P=.006).

Discussion

Main Findings

AI platforms are becoming increasingly accessible and are frequently used to obtain medical information and assist with shared medical decision-making. In this study, there was no significant difference in the comprehensiveness and clarity between the AI platforms, suggesting that both platforms provided a similar quality of AF information.

When comparing responses from cardiologists and nonmedical professionals, several key differences emerged. First, the cardiology group perceived significantly more bias in the AI-generated responses and significantly fewer explanations for the consequences of forgoing treatment. The nonmedical group rated the AI responses higher for identification and implementation of patient actions. Finally, the nonmedical group gave higher ratings for using shorter sections, informative headers, and visual cues.

Comparison With Other Studies

Overall, the cardiologist cohort assessed AI-generated AF information as accurate, yet incomplete and unclear for implementation. These findings align with prior research in other medical specialties. For example, studies regarding cirrhosis, hepatocellular cancer, and hand surgeries have shown similar findings where ChatGPT-generated prompts showed high accuracy and lower comprehensiveness [20,21].

Additionally, there was no difference in the quality of information provided between ChatGPT and Google Gemini. One study regarding adolescent scoliosis care showed similar applicability, comprehensiveness, and communication of AI-generated responses when reviewed by orthopedic surgeons, which aligns with the findings from the cardiologist group of this study [22]. However, there have been some papers in other specialties, particularly 1 study assessing maternal lactation information that showed a difference between multiple AI platforms, including ChatGPT and Google Gemini [23]. Further studies comparing these multiple AI platforms should be performed to determine which platforms perform best for a given medical topic.

Clinical Implications

The use of AI-generated responses could enhance patient education regarding AF. However, it is essential that these responses are comprehensive without compromising their clarity. For instance, AF education should be accessible to the average patient and include all relevant information on rate and rhythm control, anticoagulation strategies, and benefits and risks of catheter ablation. Additionally, cardiologists should review these responses for accuracy and completeness, recognizing that poor quality could adversely affect patient decision-making. Ideally, cardiologists should use high-quality AI-generated responses to supplement in-office discussions with patients.

Future Directions

One potential future direction is to evaluate whether providing AI-generated responses to patients in a cardiology clinic can improve AF management outcomes. A previous study showed that AI-assisted decision-making significantly improved decision quality and patient outcomes for osteoarthritis in an orthopedic clinic [24]. Similar research could help refine medical information provided by ChatGPT and Google Gemini. The addition of other, more medically focused AI platforms, such as OpenEvidence, could be used to test the accuracy and comprehensiveness of future responses from ChatGPT and Google Gemini.

As a continuation of this study, it could be beneficial to assess AI’s adaptability by asking it to refine the responses to address potential biases in health literacy and to provide a clearer structure. The improved prompt can be compared with the original prompt from each platform to determine whether AI was able to make the necessary improvements.

Limitations

This study has several limitations that should be addressed in future research. First, the sample size in each group—particularly among cardiologists—was modest, which may have limited the ability to detect smaller but meaningful differences [25]. Larger and more diverse cohorts could improve both the reliability and generalizability of future findings.

Additionally, each AI-generated response was labeled by platform (ChatGPT vs Google Gemini), which may have introduced preferential bias based on participants’ prior familiarity or perceptions. Presenting these responses anonymously could help mitigate this bias.

Second, the PEMAT-P and DISCERN criteria were originally developed to assess static, printed patient education materials. Therefore, their application to interactive AI-generated content may have been limited. In this study, we evaluated only the initial response from each AI platform, treating it as printed educational content a patient might find in their physician’s office. However, future research should consider developing or adapting evaluation criteria that reflect the dynamic and conversational nature of AI interactions.

Third, the study evaluated only 2 general-purpose AI platforms. Including additional tools, such as Microsoft Copilot or more clinically oriented systems such as OpenEvidence, could offer a more comprehensive assessment of AI-generated medical information. Our decision to limit the study to ChatGPT and Google Gemini was based on the need to reduce participant burden and optimize survey completion.

Fourth, we examined responses from 2 groups: cardiologists and nonmedical professionals. Including a broader range of medical professionals—such as internal medicine residents, general practitioners (in both inpatient and outpatient settings), cardiac nurse practitioners, and physician assistants—could reveal more nuanced insights into how AI-generated explanations are interpreted across clinical roles. Likewise, expanding the layperson cohort to include individuals without a college education may provide greater insight into gaps in health literacy and digital comprehension. Additionally, including participants who speak English as a second language could improve comprehensibility and accessibility of AI-generated responses across diverse populations.

Finally, this study focused only on the initial output provided by each AI platform. Future studies could explore the adaptability of conversational AI by prompting platforms to revise and improve their explanations in response to feedback. Comparing these revised responses to the originals could help evaluate whether AI tools are capable of meaningfully refining their content to improve clarity, reduce bias, and better meet patients’ informational needs.

Conclusions

AI platforms, such as Google Gemini and ChatGPT, are increasingly used as resources for medical information due to their ability to provide personalized responses. While both platforms generate information of similar quality, their responses must be optimized to enhance patient understanding and improve the presentation of medical information. This pilot study shows the potential gaps between cardiologists’ and patients’ perceptions of AF education. Further collaboration between cardiologists and AI software companies could help deliver peer-reviewed, widely accepted medical standards to patients, thereby supporting cardiologists in shared decision-making.